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ABSTRACT 


An interconnection network ii: which a control plane and 
data planes are separate and the bandwidth is extendable, 
and a melhod for transferring dala includes a control plane 
including control routers connected to the processors, for 
exchanging control information and generating information 
on sotting a palli I >r a mcssag ti msmiltcd between the 
processors, and a control line for connecting the control 
routers together in a predetermined topology, and one or 
more data planes each including dala routers each connected 
lo the processors, for transmit l inn, and receiving messages 
using the path information on messages generated by the 
control router, and a dala transfer line for connecting the data 
routers together in the same topology as that of the control 
plane. It is possible to arbitrarily extend the bandwidth of 
only the data plane, since the control plaDe and the data 
plane are separate, and to construct a high speed intercon- 
nection network since the latency time consumed when the 
path is traversed is independent of the size and frequency of 
the data transferred across the interconnection network. 
Therefore, it is possible to construct a flexible multicom- 
puter system, by taking advantage of the large bandwidth 
and short latency lime. 

7 Claims, 7 Drawing Sheets 
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FIG. 4B 
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INTERCONNECTION NETWORK 
EXTENDABLE BANDWIDTH AND METHOD 
OF TRANSFERRING DATA THEREIN 

CLAIM OF PRIORITY 

This application makes reference to, incorporates [lie 
same herein, and claims all benefits accruing under 35 
U.S.C. §11.9 from an application entitled Interconnection 
Network Cxtei d c I in h h \nd Method of Transfer- 
ring Data Therein earlier filed in the Korean Industrial 
Properly Office on Jan. 27, l')<)7, anil there dulv assigned 
Serial No. 97-2281 by that Office. 

BACKGROUND OF THE INVENTION 

field of the Invention 


The present 

work in a multicomputer system, and more particularly, to an 
interconnection network in which a control plane and a data 
plane are included and the bandwidth of the data plane is 
extendable and a method of transferring data in the inter 
connection network. 

DESCRIPTION OF RELATED ART 

The performance of a Von Nuemann computer is limited, 
due to structural problems and physical restrictions. Various 
computers are interconnected to ailow parallel processing in 
order to overcome such restrictions. Namely, a plurality of 
single processors or a plurality of symmetric multiproces- 
sors (SMP) are connected to process in parallel and to realize 
a high performance computer system. A system using this 
multicomputer parallel processing method includes a pre- 
determined number of node i i tin connection i 
for a message transfer between the nodes. 

In this case, the overall performance of the system is 
dependent on the performance of each node and the perfor- 
mance of the interconnection network which connects the 
nodes. The performance of the interconnection network is 
quantified by the bandwidth between the nodes and a latency 
lime. Therefore, in oider to transmit grt 1 in mil if data 
from a source node to a destination neide in a short time, the 
interconnection network must provide a Large bandwidth and 
a short latency time. In particular, new fields such as 
decision support systems (DSS) and multimedia systems 
require high amounts of data to be transferred between 
nodes. These applications create a need for improving the 
performance of interconnection networks. 

In an interconnection network and nodes in a multicom- . 
puter system, the interconnection network provides a path 
through which the nodes can communicate with each other. 
They can have various structures such as a ring, a Banyan, 
a tree, a hypcrcube, a mesh, and a torus. The above- 
mentioned structures each have advantages and disadvan- , 
tages. Common characteristics of earlier interconnection 
networks will be described with reference to the mesh 

In a network having a two-dimensional mesh structure, 
each crossing point denotes a node. The node includes a < 
computing unit (CU) and a router. '1 he router is connected to 
other adjeeem nodes and terms part of the interconnection 
network when the interconnection network and the CU are 
considered as dependent elements. 

A path must first be set when a message is to be trans- t 
ferred between nodes. An appropriate method selected from 
among a circuit switching method, a packet switching 


method, a wormhole routing method, and virtual cut-through 
method can be usee; for setting the path. In general, the path 
setting method is determined when the router is designed. 
Various matters such as the end use of the system, the size 

s of messages, and a transfer frequency must be fully consid- 
ered before Ihe method for setting the path is selected, since 
setting the path is important in determining the latency time 
of the interconnection network. 

In general, a router needs to exchange information of a 

U special purpose with other adjacent routers, lor selling paths, 
managing the interconnection network, and detecting net- 
work errors. In an earlier interconnection network, such 
information shares the same link of the interconnection 
network with the general data. However, the link of the 

5 interconnection network can not be effectively constructed 
in such a situation. Namely, in the earlier interconnection 
network, when large amounts of data must often be 
transferred, a control signal for setting the path or managing 
the interconnection network shares the link of the network 

0 with data to be transferred, thus inevitably causing interfer- 
ence between them. Accordingly, the effective bandwidth 
and the latency time deteriorate. Also, the design of the 
route: becomes complicated and Ihe lime required for deter- 
mining the path becomes longer, since the control signal and 

, the transfer data must he simultaneously processed. 

Furthermore, the bandwidth of the link between routers is 
determined when the router is designed in the interconnec- 
tion network in the earlier interconnection network such as 
BYNET of NCR, the SERVERNET of Tandem, and the HPS 

0 of IBM. Therefore, in case the bandwidth and latency time 
iu la in room cti nn re t 1 mus In improved in order 10 
be used for a next generation application, earlier intercon- 
nection networks do not provide a convenient mechanism 
lor making the improvement. 

s hi general, in order to increase the bandwidth, high speed 
semiconductor technology must used or the number of pins 
of a semiconductor chip must be increased. However, such 
methods are costly, and, in many cases, the uppermost limit 
of semiconductor integration technology has already been 

» reached. A method of overlapping multiple interconnection 
networks in parallel is an alternative Tor increasing the 
effective overall bandwidth. It is possible to increase the 
throughput of standard data simultaneously transferable 
through the interconnection network by this method. 

! However, this fails to gain various advantages which can be 
obtained by increasing the bandwidth of the link in the 
router. Namely, in the case of using two overlapped inter- 
conneclion networks, it is possible In simultaneously trans- 
fer two messages, however, it is impossible to reduce the 

1 tunc required for transferring a message. 

In the case or overlapping the interconnection networks 
according to earlier technology as compared was a case of 
increasing the bandwidth at the link, assume the size of a 
message to be sent is M and bandwidth of the earlier 

t interconnection network is W. The message transmission 
time becomes M/W since either one network or another 
network is select I in the ca i independent intercon- 
nection networks. In the case of doubling Ihe bandwidth, it 
is possible to reduce the message transmission time to 

i M/2W. Therefore, an important reason to overlap intercon- 
nection networks is actually to provide error tolerance 
through redundancy rather than to extend the bandwidth. 
Also, the router must be newly designed in order to extend 
the bandwidth sii he earli item, lion network can 

. not generally provide the effect of doubling the bandwidth. 
In the earlier interconnection network, when the header of 
the message is blocked by other messages, the latency time 
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becomes longer due to the processes for setting the path 
since other nodes car, not sel paths until the source and 
destination nodes release the blocking path. 

The following patents each disclose features in common 
with the present invention: 5 

U.S. Pat. No. 4,706,150 to Lcbizay ct al, entitled a 
tching 1> rot i it i j iv Sni $ 

Planes, U.S. Pat. No. 5,430,442 to Kaiser et al, entitled a 
Cross Point Switch with Distributed Control, U.S. Pat. No. 
5,444,700 io Miulikainen el al, entitled a Switching Moment 10 
and Method for Controlling the Same, U.S. Pat. No. 5,377, 
333 to Nakagoshi ct al, entitled a Parallel Processor System 
Having Computing Clusters and Auxiliary Clusters Con- 
nected with Network of Partial Networks and Exchanges, 
U.S. Pat. No. 5,396,231 to Itein, entitled a Modular Com- 15 
munications Interconnection, U.S. Pat. No. 5,617,539 to 
i t utiileil ./ M i \ 

with .Separate Data Network and A / V Nelw< irk Controlled by 
i i sn irtm i the Data Network, U.S. Pat. 
No. 5,659,784 to Inaba et al, entitled a Multi-processor 
System Having Convmtnication Register Modules Using 
Test-and-Set Request Operation for Synchronizing 
Communications, U.S. Pat. No. 5,603,044 to Annapareddy 
et al, entitled an Interconnection Network fur a Multi-nodal 
Data Processing System Which Exhibits Incremental 25 
Scalability, U.S. Pat. No. 5,689,722 to Swarztratiber, 
entitled a Multipipeline Multiprocessor System, U.S. Pat. 
No. 5,682,479 to Newhall et al, entitled a System and 
Method for Network Exploration and Access, U.S. Pat. No. 
5,598,408 to Nickolls et al, entitled a Scalable Processor to 30 
Processor and Processor to I/O Interconnection Network 
and Method for Parallel Processing Arrays, U.S. Pat. No. 
4,929,939 to Varma et al, entitled a High-speed 1 v\ chit i> 
System with flexible Protocol Capabib \ I S I'd Jo 
5,701,416 to Thorson et il milled " \ J i 
Mechanism for Torus Interconnection Network, and U.S. 
Pat. No. 5,680,634 to Estes, entitled a fixed Interconnection 
Network Method and Apparatus for a Modular Mixed- 
resolution, N-dimensional Configuration Control Mecha- 


SUMMARY OF THE INVENTION 

n object of the present invention to provide an 
i network in a multicomputer system, in 45 
which the bandwidth in the interconnection network is 
extendable and the same latency time is required regardless 
of the size and frequency of data, where a path is set by 
additionally including a control plane and a data plane in an 
interconnection network and the bandwidth of the data ,,, 
network is extendable. 

It is another object of the present invention to provide a 
method for transferring data in the above noted intercon- 
nection network. 

To achieve the first object, there is provided an intercon- 55 
ncclion network whose bandwidth is extendable, for trans- 
mitting and receiving messages between a predetermined 
number of processors constituting a multicomputer system, 
comprising a control plane comprising control routers con- 
nected to the processors, for exchanging control information 60 
and genciating information on setting a path for a message 
transmitted between the processors, and a control line for 
connecting (he conlrol routers together in a predetermined 
topology, and one or more data planes each comprising data 
routers each connected to the processors, for transmitting 65 
and receiving messages using the path information on mes- 
sages generated by the control router, and a data transfer line 


for connecting the data routers togc titer in the same topology 
as that of the control plane. 

To achieve the second object, there is provided a method 
of transferring data in an iuleicunuection network compris- 
ing a control plane and one or more data planes having the 
same topology as that of the conlrol plane, comprising the 
steps of transferring header information having the address 
of a destination node, from a source node to the destination 
node along a path of intermediate nodes according to the 
topologv md establishing a path for trai mining a message 
i cross the control plain, lai herring in matiou on the path 
across the control plane to the data planes, and establishing 
a path for transmitting a message across the data planes, 
releasing the established path across the control plane in 
order to 1 1 in nut u n i ige, m li nsraitting lata to 
the destination node along the established path across the 
data planes. 

BRIEF DESCRIP TION OF THE DRAWINGS 


Amore complete appresi u t i n\ ntion and many 
of the attendant advantages thereof, will be readily apparent 
as the same becomes better understood by reference to the 
following detailed description when considered in conjunc- 
tion with the accompanying drawings in which like refer- 
ence symbols indicate the same or similar components. 


FIG. 1 shows the structure of an interconnection network 
and a node, in a multicomputer system; 

FIG. 2 and FIG. 2A together show a mesh plan 
XX n u" to > r I is tcchnolog) 

FIG. 3 compares the case of overlapping ir 
networks according to earlier technology with the case of 
increasing the bandwidth of a link; 

FIG. 4A through 4C show message transferral processes 
in a mesh plane according to earlier technology; 

FIG. 5 shows an interconnection network having a one- 
dimensional structure according to the present invention; 

FIG. 6 is a flowchart showing a data transferral method 
according to the present invention; 

FJG. 7 shows a two-dimensional mesh plane structure as 
an embodiment of the present invention; 

FIG. 8A through 8C show message transferral processes 
in the mesh plane according to the present invention; and 

FIG. 9 shows a structure in which the bandwidth of a data 
plane is tripled m the interconnection network according to 
the present invention. 


Hereinafter, the present invention will be described in 
detail with reference to the attached drawings. 

In the present invention, data transmitted between nodes 
and control signals required by the interconnection network 
in order to transmit the data are each carried by a physically 
separate nelwork. The network by which the control signals 
are transmitted is called a control plane, and the network by 
which dala is transferred is called a data plane 

FIG. 1 shows tl enci I in I inlerconnc lion 
network and node in i am ti impul stem as noted above 
in the Description of the Related Art while FIG. 2 and FIG. 
2A together show a network having 2-dimensional structure, 
each crossing point n I ng node, a- also noted above in 
the Description of the Related Art. 

In FIG. 3, the case of overlapping the interconnection 
networks according to earlier technology is compared with 
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the case of increasing the bandwidth of a link. Assume the 
size of a message to be sent is M and the bandwidth of the 
earlier intcrcooucctioa network > W. Message transmission 
time becomes M/W since either the net 1 or the net 2 is 
selected in the case of two independent interconnection 
networks. Meanwhile, in the case of doubling the 
bandwidth, it is possible to reduce the message transmission 
time to M/2W. Therefore, an important reason to overlap 
interconnection networks is actually to provide error toler- 
ance through redundancy rather than to extend the band- 
width. Also, the router must be newly designed in order to 
extend the bandwidth, since the earlier interconnection net- 
work cannot generally provide the effect of doubling the 
bandwidth. 

I i I ii 

connection rictw. k^ evil Ix iVsj bed with reference t i the 
attached drawings flu inter mm tion r twork is assumed 
to have a mesh structure. FIGS. 4A through 4C show 
message iransfcrrfl processes in a mesh plane according in 
earlier technology. In FIGS. 4A through 4C, A, li, and (' 
denote source nudes and A', 11', and C denote corresponding 
destination nodes. Any method among the circuit switching 
method, the packet switching ncl od md the wornholc 
routing method can be used. First, in FIG. 4A, the path of a 
message has been sot from A to A' and data is being 
transferred. B and C are in the process of setting the path. Al 
this lime, since A [ready u part of tl path desired to be 
used by R arid C, the header of the message transferred by 
B and the header of the message transferred by C must stand 
by in the (1,1) node and the (2,1) node, respectively, until A 
completes the transfer of the message and releases the path 
which is used. When A completes the transfer of the 
message, the message header of 1? which stood by in the 
(1,1) node and the message header of C which stood by in 
I he (2,1) node can proceed to set the padis to the respective 
..imi ion u les.HG.41! hows such processes Ys shown 
in FIG. 4C, B and 0 completed setting the paths and can 
transfer message t h wtive destination nodes a)c i_ 
the set paths. Therefore, when the header of a message is 
blocked by other messages, in the earlier interconnection 
network, the latency time becomes longer due to the pro- 
cesses for setting the path since other nodes cannot set paths 
until the source and destination nodes release the blocking 
palh. 

As shown in FIG. 5, an interconnection network accord- 
ing to the present invention is separated into the control 
plane and the data plane. The control plane is connected to 
the processors which constitute the nodes through interlace 
adapters 570, 572 and 574, and includes control routers 510, 
512 and 514 one per node for exchanging control informa- 
tion for setting the path of messages, managing the inter- 
connection network, and detecting errors. The. control plane 
also includes a control line 540 for connecting the control 
routers by a predetermined topology. The data plane is 
connected to the processoi hi I stitute the nodes 
through the interface adapters 570, 572 and 574, and 
includes data routers 520, 522 and 524 one per node for 
setting the path of messages transferred between the pro- 
cessors IU lata pi in Iso includes a data transfer line 550 
for connecting the data routers 520, 522 and 524 by the same 
topology as that of the control plane. 

The data plane is extendable to a predetermined number 
of planes which is more than one, FIG. 5 shows an extend- 
able data plane #2 (iuclud utci 53(1 532 and 534 
and a data transfer line 560) and shows that the number of 
data planes is exte * ih .vith n a limit imposed by the 
interface adapters 570, 572, and 574. 

FIG. 6 is a flowchart shewing a method for transferring 
data according to the present invention. According to the 


present invention, the path from the source node to the 
destination node is determined on the control plane (step 
Mif Namel header sent Irom tl ue e in lueles 
the address of the destination node. The node which receives 
the heaele; determines its path based on the destination node 
address. When the path has been determined on the control 
plane, each control plane router on the path gives path 
information to the corresponding router on Ihe data plane 
(step 610). In this way, a path through which data can be 
transferred from the source node to the destination node is 

to constructed on th a [ Wh u tl processes >f setting 
the path are completed, the header is transferred toward the 
destination node on the control plane and data horizontally 
moves along the header on the data planes (the first data 
plane, the second data plane, . . . ). Therefore, the present 

is invention is called a Tornado routing method (step 620). 
The prist nt n 1 nti nea.nb tppl I to irious topologies 
such as a ring, a Banyan, a tree, a hypercube, a mesh, and 
a torus. However, in an embodiment of the present 
invention, an application lo a two dimensional mesh plane 
structure will be described. 

20 FIG 7 shows the tw intension sh plane structure 
according to the present invention. In Ihe two dimensional 
mesh plane stru i 4 wa in FIG. 7 thin lines denote the 
conlrol plane and thick lines denote the data plane. In FIG. 
7, the data plane uses only one plane. However, the physr- 

2$ e'ally separate data plane is extendable as shown in FIG. 9. 
In the present invention, Ihe control plane and the data plane 
use different routers. They are respectively called control 
routers and data routers. 
FIGS. 8A through 8C show processes of transferring a 

, f) it « gt n t ii s) pl nit rdii It) the present cr 
lion. In FIG. 8A, the message from A is already being 
transferred along a set palh, and the messages from B and C 
must use nodes on the path occupied by the message of A. 
In the present invention, since the data plane is separated 
from the control plane and the palh is set in the control plane, 

- 5 ihe messages of B and C can proceed to set paths regardless 
of the path of message A on the data plane. As soon as Ihe 
control rnulcrs se! the: path on the control plane, the path 
information is transferred to the data router corresponding to 
the same node. The data router can then begin to transfer the 

*) message along the path determined by the control plane, 
using the path information. FIG. 8B shows a ease in which 
the paths of the messages of R and ( ' have been set, but the 
message o I Vis lilll I t th nth, ['he messages of B 
and C can be transferred lo the nudes (1,1) and (2,1) until the 

45 path of A is released. Then, since the paths of B and C are 
already set. the messages can be transferred as soon as A 
releases the path, without a delay to set the paths of B and 
C, as shown in FIG. 8C. 

FIG. y snows an embodiment in which the number of data 

50 planes is extended by two in older to triple the bandwidth. 
It is noted thai the bandwidth of the interconnection network- 
is theoretically extendable. According to the present 
invention, it is possible to arbitrarily extend the bandwidth 
of only the data plane, since the control plane and the data 
plan 1 -'' arc separated in the interconnection network, and to 
construct a high speed interconnection network, since the 
latency time consumed when the path is set is independent 
of the size and frequency of the data which is transferred 
across the intc re a ct i net irl therefore, it is possible 
lo construct a flexible multicomputer system, taking aelvan- 
«> tagc of the large bandwidth and short latency time. 
What is claimed is: 

1. An interconnection network for a multiprocessor com- 
munication system, said interconnection network compris- 
ing a plurality of nodes arranged in a predetermined topo- 
65 graphical structure, each of said nodes comprising: 

a single interlace connected to a corresponding single 
processor for receiving source message data from said 


US 6,304,568 Bl 


processor and transmitting destination message data to 
said processor; 
a control router connected to receive control data from 
said interface; 

a first data router connected to receive and transmit 5 
message data over a first data connection with said 


a second data muter connected lo receive and transmit 
message dala over a second data connection with said 
interface; 

a control line connected to each of said control routers of 
each of sa:'d nodes for transferring said control da'a 
from a control router of a source node to a control 
router of a destination node, said control line and said , 
control routers forming a control plane arranged in said 
predetermined topographical structure, 
a first data line connected to each of said first data routers 
of each of said nodes for transfer) ne in < e da 
from a first data router of said source node to a first data : 
rotiler of said destination node, said first data line ard 
s id irs data i niters foi u first da I plan irranged 
in said prcdeteri t | hi tructure ind 

a second data line connected to each of said second data 
routers of each of said nodes for transferring message : 
data from a second data router of said source node to a 
second data router of said destination node, said second 
data line and .said second data n alters forming a second 
data plane arranged in said predetermined topographi- 
cal structure, wherein said first and second data lines ? 
cany only message data and said control line carries 
only control data. 

- i i nk eoi n 11. von-' inn' proci or cot 

muuicaliun system as set forth in claim 1, wherein each said 
interface of each of said nodes comprises further data 3 
connections for connecting to further data routers of said 
node to extend a bandwidth of said interconnection network. 

3. 'Ihc interconnection network for a multiprocessor com- 
munication system as set forth in claim 1, wherein prede- 
termined topographical structure is ring, a Banyan, a tree, a 4 
hypercube, a mesh or a torus iiitercoriTieetion network. 

4. The interconnection network for a multiprocessor com- 
munication system as set forth in claim 1, wherein said 
control data comprises header information including an 
address of said destination node, and said control router of 4 
said source node utilizi 5 said 1 Iress lo set .- message data 
communication padi through intermediate nodes to said 
destination node ove-onc of said first and second data pianos 
and to set a control data communication path through said 
imcmiediale nodes lo said destination node over said control 
plane. 5. 

5. The interconnection network lor a multiprocessor com- 
munication system as set forth in claim 4, wherein each of 
said control routers of said intermediate nodes in said 
communication path determine whether a data router in said 
message data communk lion path is Iree to transmit mcs- 5: 
sage data, such that Ihc control router, which determines that 
a corresponding data router is not lree, blocks the transmis- 
sion of said message data until it is determined that said data 
router is free, said message data communication path 
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6. A method of transferring data in an interconnection 
network comprising a plurality of nodes arranged in a 
predetermined 1 pograpl il ruelure, each o said nodes 
comprising a single processor for transmitting and receiving rt 
control data and message data, a contn 1 router dis] ed in v 
a control plane, one or more data routers, arid one or more 


data planes in each of which one of the data routers is 
disposed, respectively, said method comprising the steps of: 
detecting header information in control data transmitted 
from a source processor in a source node and deter- 
mining destination processor a destination node in 
response lo address data in said header information; 
selling a control data communication path through inter- 
mediate nodes to said destination node over said con- 
trol plane and setting t separate nessagi data commu- 
nication path ihi trgl id same ilermediale nodes to 
said destination node over one of said data planes; 
transferring only said control data from the control router 
of said source node lo each control router in said 
control data communication path; 
determining, b e 1 of said < nitrol routers in said control 
data communication path, whether a corresponding 
data router in said message data communication path is 
free to transmit message data through the correspond 
node wherein 1 me sag data o nmunication path 
transmits only message data; 
blocking the message data communication path to prevent 
transmission of said message data, when it is deter- 
mined that one of said data routers is not free, while 
maintaining said control data communication path; 
releasing the message data communication path, when it 
is determined thai said one of said data routers is free, 
lo transmit said message data to the processor of the 
destination node. 
7. A method of transferring data in an interconnection 
network comprising a plurality of nodes arranged in a 
predetermined topographical structure, each of said nodes 
comprising a single processor for transmitting and receiving 
control data and message data, a control router disposed in 
a control plane, a first data router disposed in a first data 
plane and a second data router disposed in a second data 
plane, said method comprising the steps of: 
detecting header information in control data transmitted 
from a source processor in a source node and deter- 
mining destination processor a destination node in 
response to address data in said header information; 
setting a control data communication path through inter- 
mediate nodes to said destination node over said con- 
trol plane and setting a separate message data commu- 
nication path through said same intermediate nodes to 
said destination male over one of said first and second 
data planes; 

transferring only said control data from the control router 
of said source node to each control router in said 
control data communication path; 

determining, by each of said control routers in said control 
da'a communi n path bethel e S| ling 
data router in said message data communication path is 
free to transmit message data through the correspond 
node; wherein said message data communication path 
transmits only message data; 

blocking the message data commit tticatiou path to prevent 
transmission of said message data, when it is deter- 
in met! that one of said data routers is not free, while 
maintaining said control data communication path; 

releasing the m ; < 1 mmiini 1(1 n path when il 
is determined that said one of said date routers is free, 
to transmit said message data to the processor of the 
destination node. 


